I need a airline crawler where is possible to do 1000 scans (10 origins for 100 destinations for example) in less than 30 seconds in a server like "amazon aws ec2 [login to view URL]"
The solution should preferably be in nodejs, but I accept other sujings if it proves to be more efficient in computational cost
The input parameters are:
${origin}
${destination}
${leaveDateYmd}
${returnDateYmd}
The link format is:
[login to view URL]${leaveDateYmd}0000&B_LOCATION_1=${origin}&E_LOCATION_1=${destination}&LANGUAGE=BR&SITE=JJBKJJBK&TRIP_TYPE=R&WDS_MARKET=BR&adults=1&children=0&infants=0&COMMERCIAL_FARE_FAMILY_1=NEWBUNDLE&B_DATE_2=${returnDateYmd}0000&utm_source=zanox&utm_medium=affiliate&utm_campaign=metabuscador
Result Link example:
[login to view URL]
I will give a list of origin and destination for test
The result estructure:
var resultData =
{
adults: (For now you can leave as 1)
airline: (LATAM),
leaveDate: (input Leave Date),
returnDate: (input return Date),
//Vector of leave data
{
price: (scanned Price),
tax: (scanned tax)
totalDuration: (scanned TotalDuration
//The parameters below will be vectors, since a trip may have 1 or more aircraft (as there may be a connection)
{
flightNumber: (scanned flight Number),
origin: (scanned origin ),
destination: (scanned destination ),
stops: (scanned number of stops),
depTime: (scanned departure Time "hh:mm"),
arrTime: (scanned arrive Time "hh:mm"),
duration: (scanned duration),
connectionDuration: (scanned time in connection)
}
}
//Vector of return data is the same of leave data
{
}
}