We have a working MySQL db which has a table that receives up to 1000 new records per minute. We need to make a script that periodically reads the data, filters it for duplicates and process some of the timestamps before writing the data into another PostgreSQL table on another server.
All this needs to be done with the smallest delay possible between the data being written into MySQL and the data being processed and copied to PostgreSQL. We have to avoid duplicate records in the target PostgreSQL db at all costs.
I am looking for people with experience in processing large amounts of data in a relatively short time and who can build the system as close to realtime syncing as possible. I am open to all design suggestions and can deliver sample databases for the successful bidder to start work on.