Find Jobs
Hire Freelancers

JSON to MySql using Python

$30-250 USD

Completed
Posted about 8 years ago

$30-250 USD

Paid on delivery
I need a Python multithreaded script to download json from a remote url (API). It then processes the data accordingly and writes the data to MySQL. I already have a php script written that contains all of the business logic and that is running as a cron job on a GoDaddy Linux shared hosting account. It works, but due to scalability concerns, I want to run & evaluate this as a multithreaded Python script. I'll collect bids until Monday, April 18th before making a decision. The business logic portion of this script is approximately 100 lines (~40 more lines in debugging/var_dump helpers). The purpose of the script is to maintain a list of users that were online, when they got online, and when they logged off. The script uses a remote web API call to get a list of users currently online. Here's a basic overview of the logic: 1. Select MySQL users that exist in local db in 'online' 2. Uses cURL to fetch new JSON data 3. For each user in JSON, store each user not 'online' in 'newOnline'. 4. For each user in 'online' not in JSON, store as 'offline'. 5. Update MySQL for 'offline' users. 6. Insert MySQL for 'newOffline' users. 7. Insert MySQL json data (for future reference). 8. Use the keyword "timeclock" in your response to indicate you read this project description.
Project ID: 10238232

About the project

20 proposals
Remote project
Active 8 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
Timeclock Hello, I am confident I can complete this project for you within a few hours and could start straight away - however I appreciate that the description states you will collect bids until the 18th. Would the Python script need to be in 2.x or 3? I look forward to hearing from you. Regards, Josh.
$100 USD in 1 day
5.0 (29 reviews)
4.7
4.7
20 freelancers are bidding on average $144 USD for this job
User Avatar
I'm Python developer with many years of experience and I know about "timeclock" that's why I'm sure you'll be impressed with my work. I can update your code (convert it from PHP to Python?) in couple of days and I can offer you best price here. All I need to start is your current code and milestone payment from you. You'll release it after you check everything on your side. Thanks. Roman
$168 USD in 2 days
4.9 (542 reviews)
7.7
7.7
User Avatar
Hi there, I have read the project & would like to discuss.. I have good web scraping skills in Python & have worked with APIs, JSON and MYSQL in Python.. Hope to hear from you..
$90 USD in 1 day
4.9 (105 reviews)
6.2
6.2
User Avatar
I'm qualified for this job..........................................................................................................................................................
$94 USD in 3 days
5.0 (35 reviews)
6.0
6.0
User Avatar
A proposal has not yet been provided
$100 USD in 3 days
4.8 (83 reviews)
6.2
6.2
User Avatar
timeclock Hi. I can do it with python gevent/greenlet or just with multiprocessing or with grab library. But I suppose bottleneck will be in mysql. For this we can use memory engine in mysql. How many users you want monitor?
$188 USD in 3 days
5.0 (41 reviews)
5.3
5.3
User Avatar
timeclock I have built many scrapers in python amd have stored data to mysql as well. I would use twisted framework for asynchronous processing because the code is waiting on external resources (remote API, database) most of the time. I've used that approach several times and that's how popular Scrapy framework works.
$111 USD in 2 days
5.0 (19 reviews)
5.3
5.3
User Avatar
Let me help you ************************ ********************* ********************************** *************************************
$100 USD in 10 days
5.0 (11 reviews)
5.1
5.1
User Avatar
I'm experienced in this part, i will do it quickly. Hire me, i will get you the best. Thanks, TcBinh
$50 USD in 5 days
4.9 (9 reviews)
4.9
4.9
User Avatar
A proposal has not yet been provided
$88 USD in 2 days
5.0 (15 reviews)
4.2
4.2
User Avatar
timeclock Dear sir, Thank you for posting this project. I have read and understand project logic. Some questions need clearly: - How often you want script to be run? every min? - How many records for each query to Web API (average)? - And very important question, is API return defference data for each seperated query (I mean for 2 very closed time query)? Because, as logic "get new offline user", if seem multhreads will not increase performent of system. It's dependence data for query in order, so parallel maybe cause problem. Waiting more discuss from you, Hung
$45 USD in 2 days
5.0 (26 reviews)
4.3
4.3
User Avatar
Dear employer, I am experienced python programmer, and could build this as multi-threaded python script. Considering your info, regular python multi-threaded script would be enough but have in mind that full scalability can be achieved by having Que and Celery (workers, beat). Yours sincerely, Ivan - >
$122 USD in 3 days
5.0 (21 reviews)
4.1
4.1
User Avatar
send me a message so we can discuss about your requirements. thanks.
$111 USD in 3 days
4.9 (17 reviews)
4.2
4.2
User Avatar
Chat now. 7 yrs experienced NZ developer; new on freelance for global clients. Many projects are on my site. Most of its components are already with me. I am online so plz contact me now and get most of it built within a day. you can pay me after you work.
$494 USD in 3 days
5.0 (4 reviews)
3.0
3.0
User Avatar
I written python script which were used for store data to mysql database. In that data came from API response.I done projects in python-django in animation, e-commerce domain. We deployed server to aws server with apache. I used some python and django package like django-cms, django-celery(for cronjob/automation), django-restframework(for api). Basically I am a full stack developer.
$166 USD in 3 days
5.0 (1 review)
0.6
0.6
User Avatar
You can contact with me at any moment to discuss your task You can contact with me at any moment to discuss your task
$250 USD in 5 days
0.0 (0 reviews)
0.0
0.0
User Avatar
A proposal has not yet been provided
$138 USD in 12 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of UNITED STATES
Tampa, United States
5.0
3
Payment method verified
Member since Apr 16, 2016

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759) & Freelancer Online India Private Limited (CIN U93000HR2011FTC043854)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.