Automate the setting up of Greenplum database on EC2

I am going to do some performance testing on how well Greenplum performs when it utilizes many EC2 servers to run a query. Therefore, I need you to automate the process of starting many EC2 instances, installing Greenplum on each one, and setting up Greenplum so queries can be run in parallel. The setting up of Greenplum is described in detail in their admin guide. The admin guide is found by registering and logging into: <[login to view URL]> I want the scripting to be written in python, possibly using tools like pexpect or something. The python script should run on Windows, and it would be nice if it ran on Linux. The basis EC2 instance should be ami-7c1c3408 (located in the EU). My interface would be to set the following variables in the python code: secretKey = "adfjndfndf" theotherkey = "adjdjfnjdfn" number_of_greenplum_nodes = 40 greenplum_password_to_be_used = "adfadfjndf" Possibly setting paths to various keys placed on the local machine. This should be all that is needed from my side. When I click "run" everything should be automated right up to where I can connect to the coordinator node at port 6453 (or whatever it was). You may use WinSCP and other external tools, but please include them in your final deliverable so I can just click "run" to test your solution. Your setting up of the EC2 and greenplum should be secure so hackers could not easily get in. It is OK that a [login to view URL] needs to be placed on a server where it is accessible without login and password (unlike <[login to view URL]>). It should be easy to modify your python code, so I can ask other coders to modify it to get greenplum to run in a more optimized way. You do not need to set up Greenplum with agents (I believe this is their name) on each node. ## Deliverables The following is my current semi-automatic way of doing what is needed, I cannot guarantee that it works, but it gives you an overview of the task. In this description, the EC2 instances are launched manually in elasticfox: AMI: ami-7c1c3408 (EU) useradd gpadmin echo mypassword | passwd --stdin gpadmin echo mypassword | passwd --stdin root wget [login to view URL] mv -f sshd_config /etc/ssh/ /etc/init.d/sshd restart su - source /usr/local/greenplum-db/[login to view URL] cd /usr/local gtar -cvf /home/gpadmin/[login to view URL] greenplum-db source /usr/local/greenplum-db/[login to view URL] gpscp -f /home/gpadmin/single_seg_hosts_file /home/gpadmin/[login to view URL] =:/usr/local gpssh -f /home/gpadmin/single_seg_hosts_file gtar --directory /usr/local -xvf /usr/local/[login to view URL] chown -R gpadmin /usr/local/greenplum-db chgrp -R gpadmin /usr/local/greenplum-db mkdir /mnt/data chown gpadmin /mnt/data chgrp gpadmin /mnt/data exit Or: You also need to edit /etc/ssh/sshd_config to allow root logins with passwords. The line: PermitRootLogin without-password should be changed to: PermitRootLogin yes S?k etter tekst (flere steder): PermitRootLogin og skriv PermitRootLogin yes Search for text (exists several places): PasswordAuthentication and replace with: PasswordAuthentication yes run: /etc/init.d/sshd restart chown -R gpadmin /usr/local/greenplum-db chgrp -R gpadmin /usr/local/greenplum-db source /usr/local/greenplum-db/[login to view URL] source ~/.bashrc mkdir /mnt/data chown gpadmin /mnt/data chgrp gpadmin /mnt/data create /home/gpadmin/all_hosts_file which contains a list of ipaddresses with master on the first line. See page 46 in admin guide. Create /home/gpadmin/single_seg_hosts_file which contains a list of ipaddresses withOUT master on the first line. See page 46 in admin guide. Create /home/gpadmin/multi_seg_hosts_file which is identical with single_seg_hosts_file. gpssh-exkeys -f /home/gpadmin/all_hosts_file gpssh -f /home/gpadmin/single_seg_hosts_file '/usr/sbin/useradd gpadmin -d /home/gpadmin -s /bin/bash' chown gpadmin /home/gpadmin/multi_seg_hosts_file chgrp gpadmin /home/gpadmin/multi_seg_hosts_file su - gpadmin source /usr/local/greenplum-db/[login to view URL] gpssh-exkeys -f /home/gpadmin/all_hosts_file gpssh -f /home/gpadmin/single_seg_hosts_file -v date cd /home/gpadmin/ wget [login to view URL] Modify gp_init_config with inserting the correct ipaddress for the master node. cd /home mkdir s3sync cd s3sync wget [login to view URL] gunzip [login to view URL] tar xvf [login to view URL] cd s3sync Add following text to /etc/s3conf/[login to view URL]: export AWS_CALLING_FORMAT=SUBDOMAIN /home/s3sync/s3sync/[login to view URL] get abucket:[login to view URL] /home/[login to view URL] cd /home/ unzip [login to view URL] chown gpadmin /home/[login to view URL] gpscp -f /home/gpadmin/single_seg_hosts_file /home/[login to view URL] =:/usr/local (to initialize greenplum): su gpadmin gpinitsystem -c /home/gpadmin/gp_init_config edit [login to view URL] add i.e. following line: host all gpadmin [login to view URL] md5 export MASTER_DATA_DIRECTORY=/mnt/data/gp-1 source /usr/local/greenplum-db/[login to view URL] gpssh -f /home/gpadmin/single_seg_hosts_file "echo 'work_mem = 32MB' | cat - >> /mnt/data/gp*/[login to view URL]" gpssh -f /home/gpadmin/single_seg_hosts_file "echo 'maintenance_work_mem = 64MB' | cat - >> /mnt/data/gp*/[login to view URL]" gpssh -f /home/gpadmin/single_seg_hosts_file "echo 'max_connections = 15' | cat - >> /mnt/data/gp*/[login to view URL]" export MASTER_DATA_DIRECTORY=/mnt/data/gp-1 gpstop -r (confirmation needed) psql template1 (press enter) 1) All deliverables will be considered "work made for hire" under U.S. Copyright law. Employer will receive exclusive and complete copyrights to all work purchased. (No 3rd party components unless all copyright ramifications are explained AND AGREED TO by the employer on the site per the worker's Worker Legal Agreement). ## Platform Windows/Linux/EC2