Mirror/crawl/duplicate an entire (large) website -- 2

Hi To Mirror a site, you need to download the HTML pages, scan the pages and then download from all the linked media and find all the html pages and then repeat for them. This can be done via requests and Beautiful Soup. Requests is the most popular Python library and is used to download the files. BeaultifulSoup is used to parse the html and find the links etc. If the site is large, I guess the best route would be to go through the site, scanning for all the links, storing non html links and following the html links. That will then give us a list of the media files of various sorts, that we can then download in a separate action. Since the url also specifies the directory structure, I can use that to build the structure for you to mirror., Thanks Marc Nealer

Hi, I normally don't trust projects asking to mirror website instead of parsing it, but let's see. hitting it with multiple instances won't be a problem if I'll be interested in doing this. Thanks

Dear Sir, I'm very much delighted to let you know that I've been doing data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database or excel or csv or xml file. I worked on many similar projects, I have big experience in data mining projects. I have written hundreds of web scrapers which scrape millions of pages each day. I'm ready to fulfill your requirement. I can finish this task in short time, with the best quality. I can assure 100% accuracy. Please give me the opportunity to do the work. With Kind Regards, Debdulal Roy

Hi, I have read your requirement carefully and I can do this project. I have over 7+ years of experience along with in-depth technical knowledge on Python, Flask, Node.js, JavaScript, PostgreSQL Administration, Django, MySQL, MongoDB, XML-RPC services, Data Analysis, Web scrapping, ERP/CRM portals and implement the fully innovative and creative strategy to deliver with outstanding functionalities and user experiences. I have developed a variety of Software Applications including Hospital Management System, University Management System, Inventory Management system and many others. Developed dynamic Websites, and crypto trading indexing. Specialities: Database systems(MySQL, PostgreSQL), Can Program fluently in ML, DL, AI, Python, and PHP. Proficiency in Odoo software development. Knowledge of IT Security and Assurance, Linux System Administrator, Crypto, and Binary Options Trading with more than 10x return. Extensive experience with planning to design and configuring scalable, highly available and redundant networks on Amazon Web Services cloud platforms. Message me to get more details about my relevant experience or provide info about your needs! Looking forward to hearing from you. Thanks

I am writing this proposal in order to work for you in Software and Web Development. We are highly trained professional developers seeking to freelance and earn online. Having a flair in programming and development I have been excellent with JAVA,C#,C/C++ and PHP programming language along with MySQL with XAMPP Server. I also build application for Cloud computing and High performance computing as I also have knowledge regarding Hadoop, MPI, OpenMP (distributed frameworks) and also have build applications in CUDA C++ and OpenCL. I also have experience with the most famous framework ASP.NET. My most expertise are in JAVA where I have built countless projects for semester projects and Final year project. You may find many developers in this field however we assure that you will not be able to find a team like us. We not only ensure the code is quality wise but we also assure that the code we write are optimised and we ensure that the program performs right operation under right environment i.e. we create programs that are defect free. You may also find freelancers that cost low but they do not put their 100% which then shows in the software and leads to an unhappy customer. We ensure reasonable price are put as tag on the job and we ensure in order to give the write product. We highly appreciate your time, if you are interested kindly let me know

Hi, My name is Sandeep, I am senior Mobile and Web developer with 8+ years of rich experiences. I have good skills in Python, React Native, Angular, Php and Javascript. ________________________________________________ Single point of contact to deliver end-to-end: _________________________________________________ - Custom Software and Mobile App Development - Web development, CMS based solutions, JSON, 3rd party API’s like Google maps - Business Application and Web Portals - Proficient in Platforms like NodeJs, ReactJS, React Native, Python for small and medium global customers. - PHP, Codeingnitor, HTML, CSS, Angular, jQuery, Javascript etc Our clients have come to rely on our expertise, reliability, and speed for bringing innovation to reality. Let's start the chat so that we can discuss more on the project. Looking forward to hearing from you soon. Thanks and Regards Thanks & Regards Sandeep Gupta

Hello, I'm a full-stack developer with extensive knowledge of Python with an experience of 7+ years. Here's a sample of Backend/DevOps tasks I can help you with: - Design and implement REST APIs in Flask/Postgres, documented with Swagger, with graphql support (using graphene) - Dockerise applications and deploy them to AWS ECS, setting up load balancing and auto scaling - Move infrastructures to code using Terraform. - Integrate apps in AWS EC2, RDS, S3, ElasticCache ... - Build CI/CD pipelines for testing and automation, recommending best practices like semantic versioning and changelog automation. - Implement Monitoring/Logging for Datadog and ElasticCache, including custom instrumentation for APM. - Automate existing workflows in Python Let's connect to discuss the details. Regards, Mishal

Hi There! I have more than 8-year experience in this field. Would you please share more details about the project? I am really interested to work with you for long. Best Regards, Abhishek

I'm a CEO of a Software house. We have about 30 Employees with minimum 6 years of experience. We will give you the best most accurate work in about 3 days in which we will also provide you with the demo of the website!

Hello, can you please provide those examples of some pages you said in the project description? I am an expert in web scrapping and server management, so I hope I can find a faster way to reduce download time. Thanks in advance