Perl Script - How to modify

This document summarizes the Perl code and is a HOWTO on how to make a new perl script if they no longer function correctly.
Problem Statement: In the dialog navigation, there are queries that are not already in the database such as "How do I get from Wendy's to Walmart?" or "Where is Mugshots?". The database program has the capabilities to go out to Mapquest and using already known interactions with it, retrieve the information.
The problem with this is that the layout is a known layout, but Mapquest may change this layout once in a while. If the requesting layout does not change then the displaying layout may change. The last change Mapquest did as of this writing was around 2005-02-13.
Some helpful links:
Explains the layout for linking to maps
- http://www.mapquest.com/features/main.adp?page=lf_maps
Explains the layout for linking to directions
- http://www.mapquest.com/features/main.adp?page=lf_dir
Some types of searches that are possible
- http://www.mapquest.com/maps/main.adp?formtype=search
There have always been four scripts,
- web_location
- web_location_1
- web_address
- web_map
  The first one has always been broken. By the name of the second, it appears that it's a re coding of the first anyway, so there will no more mention of the first one.
  Now the purpose of the 3 scripts are as follows (taken from perl script):
- web_location_1 - Find a place either near an address or with a blank address
- web_address - Find the address of a type of business
- web_map - Find the directions from one place to another from their addresses
  #==========================================
  # web_location_1.perl [2]
  #==========================================
  #- inputs
  # business_type = 1 ;;name or type or anything regarding what's being found
  # start_add = 2 ;;start\search area -- stands for 'start address'
  # city = 3
  # state = 4
  # zip = 5
  # number_for_out = 6 ;;number for the outfile
  # path for out = 7 ;; '/tmp/perl' or whatever
  #
  #- outputs
  # print DATA "name $placename \n";
  # print DATA "address $local_address \n";
  # print DATA "city $cityname \n";
  # print DATA "state $statename \n";
  # print DATA "telephone $placename \n";
  # print DATA "miles $mile \n";
  #
  #
  # This query looks for a place near an address and then prints out all the results. Uses the query website of http://yp.mapquest.com/main.adp.
  #
  # This has two different queries based on whether "start_add" is present or not.
  #==========================================
  # web_address.perl [3]
  #==========================================
  #
  #- inputs
  # business_name = 1 ;; name search
  # business_spe = 2 ;; business specification
  # business_type = 3 ;; type
  # start_city = 4 ;;
  # start_state = 5 ;;
  # start_zip = 6 ;;
  # ref_name = 7 ;; used only in output - as "refer_name"
  # ref_add = 8 ;; used in the "near this address" part of the search
  # number_for_out = 9 ;;number for the outfile
  # path for out = 10 ;; '/tmp/perl' or whatever
  #
  #- outputs
  # print DATA "name $location \n";
  # print DATA "specification $business_spe \n";
  # print DATA "type $business_type \n";
  # print DATA "address $local_address \n";
  # print DATA "city $cityname \n";
  # print DATA "state $statename \n";
  # print DATA "telephone $location \n";
  # print DATA "miles $mile \n";
  # print DATA "refer_name $ref_name \n";
  # print DATA "refer_add $ref_add \n";
  #
  # This query looks for a certain business type or name NEAR a given address.
  # The three inputs business_name\spe\type are all concat'd together so ONLY PUT ONE of them.
  #
  # If "ref_add" does not contain a digit as the first character, it puts "100" on the front so that it is a true address instead of just a street name.
  #
  #==========================================
  # web_map.perl [4]
  #==========================================
  #
  #-inputs
  # start_name = 1 ;;
  # start_add = 2 ;;
  # start_interest = 3 ;;
  # start_city = 4 ;;
  # start_state = 5 ;;
  # start_zip = 6 ;;
  # end_name = 7 ;;
  # end_add = 8 ;;
  # end_interest = 9 ;;
  # end_city = 10 ;;
  # end_state = 11 ;;
  # end_zip = 12 ;;
  # number_for_out = 13 ;;
  # path for out = 14 ;; '/tmp/perl' or whatever
  # #note: all inputs are self-explanatory
  #
  #-output
  # print DATA "start_name $start_name\n";
  # print DATA "start_add $start_add\n";
  #
  # print DATA "start_interest $start_interest\n";
  # print DATA "start_city $start_city\n";
  # print DATA "start_state $start_state\n";
  # print DATA "start_zip $start_zip\n";
  #
  # print DATA "end_name $end_name\n";
  # print DATA "end_add $end_add\n";
  # print DATA "end_interest $end_interest\n";
  # print DATA "end_city $end_city\n";
  # print DATA "end_state $end_state\n";
  # print DATA "end_zip $end_zip\n";
  #
  # print DATA "total_distance $distance\n";
  # print DATA "total_time $time\n";
  # print DATA "directions $count\n";
  # for ($count)
  # print DATA "$direction"
  # print DATA "$mile"
  #
  # This query attempts to find out the directions from one place to another.
  # Only one set of directions is given per output file.
  # The output file contains the standard 15 lines, then in pairs of lines, contains ($count) number of pairs.
  #==========================================
  # Sample executions:
  #==========================================
  #
  # 1) web_location -- THIS WILL NOT WORK BUT HERE IS A SAMPLE LAYOUT
  #
  # ./web_access.perl 1 "Walmart" "" "Starkville" "MS" "39759" 0 "/tmp/"
  #
  # 2) web_location_1 - find a place
  #
  # ./web_access.perl 2 "Walmart" "" "Starkville" "MS" "39759" 0 "/tmp/"
  # ./web_access.perl 2 "Foodmax" "" "Starkville" "MS" "39759" 0 "/tmp/"
  # ./web_access.perl 2 "Stromboli" "" "Starkville" "MS" "39759" 0 "/tmp/"
  #
  # 3) web_address.perl - find something near an address
  #
  # ./web_access.perl 3 "" "gas station" "" "Starkville" "MS" "39759" "" "118 Highway 12 W" 9 "/tmp/"
  # ./web_access.perl 3 "" "movie rental" "" "Starkville" "MS" "39759" "" "118 Highway 12 W" 8 "/tmp/"
  # ./web_access.perl 3 "" "food" "" "Starkville" "MS" "39759" "" "118 Hwy 82 W" "0" "/tmp/"
  #
  # 4) web_map.perl - directions
  #
  # foodmax - 118 Highway 12 W
  # stromboli's - 408 University Dr
  #
  #./web_access.perl 4 "" "118 Highway 12 W" "" "Starkville" "MS" "39759" "" "408 University Dr" "" "Starkville" "MS" "39759" 1 "/tmp/"
  #
  #
  Creating it:
  Now, the person who edits or creates a new one should either know a thing or two about perl or be able to figure it out. This Info file will not cover anything (or very little) about the perl syntax.
  There are two functions for each type of request, and a few "helper" functions that are useful for all the requests.
  First let's cover the helper functions.
  sub find_popular_matching:
  - Sometimes when retrieving a page, the type of business is vague so Mapquest will reply with some suggestions. An example is searching for businesses of type "gas" near an address. If this still works, you can try this link: http://www.mapquest.com/maps/map.adp?formtype=search&searchtype=search&country=US&addtohistory=&cat=gas&1ahXX=&address=522+University+Dr&city=Starkville&state=MS&zipcode=39759
  - Therefore this function is safe and should be called anytime a new request is downloaded.
  It will recusively look for "Most Popular Matching Categ" and then follow and download the first link after that.
  [More Detail: The link must be off the page map.adp. Previously Mapquest used main.adp, so if it changes it again, it may be as simple a fix as finding out the new "name.adp"]
  - The reason it searches for "Matching Categ" is because it may be singular or plural "Category\Categories".
  sub find_next_page:
  - When retrieving addresses for large searches such as "gas stations", there are multiple pages of requests. Therefore this function be called and it looks for "Next" as a hyperlink, follows it and returns the new page.
  - When changing this function at all, the page must explicitly have "Next" as a hyperlink. Previously the link was "Next Page" so a fix may be as simple as finding what the new text is.
  - You can try this link and look at the bottom for the "Next" written there. http://www.mapquest.com/maps/map.adp?submit=&location=%2fYVsxMFw%2b2ut%2b6n324ss1RFNtV2am%2f6Pq5Zi8tkLDJSAgAWVwl%2bZ9PsS7H6kchFwmLPpXu%2fi8LldrAT75X3DQcN1Mr1PogC1j60wrQoyXryOoaKTciUOueAkjWxkfWUWqOn5vmw3t21KEY7Lq6Z%2bTA%3d%3d&country=US&address=522%20University%20Dr&city=Starkville&state=MS&zipcode=39759%2d3386&searchtype=search&formtype=search&id=428a54bf-003b1-03234-400c330d&dist=25&skip=10<r=&sc=61
  sub find_more_specific_address:
  - When getting the address of a location, sometimes instead of searching for a specific address, just a street name is saved. An example of this is http://www.mapquest.com/maps/map.adp?formtype=search&searchtype=search&country=US&addtohistory=&cat=gas&1ahXX=&address=100+University+Dr&city=Starkville&state=MS&zipcode=39759 because the address "100 University Dr" does not exist, so it suggests using the address "[246-399] University Dr".
  - This function is safe for calling everytime because if it doesn't find the text on the page, it will return the original.
  - The key on this page is the first <"li"> tag so if it breaks, it may be as easy as finding what the new page's link is surrounded by.
  Now all the helper functions have been covered so now the document will cover the process for changing a function if Mapquest changes.
  1) Go into the function that is no longer working and uncomment all the debugging information for that function. A very small change in the layout of the document will break things. Therefore you can determine if the query is breaking in the requesting of the page or the page layout.
  2) First let's assume it is merely the page layout that is changed. a) Download a few example pages and save them using some debug information.
  b) Look at the HTML and attempt to figure out a way to narrow down the page to only the important area. The current area looks for the location of the javascript functions named "selectSortBy" and "openGoogle". Then that section may be broken down further. Be not afraid of using very specific HTML because Mapquest often does not change the layout for many months. One example very specific line that is currently working is.
  @entries = $line =~ /<td class=boldText><b>(NAME RIGHT HERE[^<]*)<\/b><\/td><\/tr>
  <\/td> <\/td> (ADDRESS HERE[^&]*) \((MILES AWAY[0-9.]*) miles away\)
  <\/td> \ <\/td> (PHONE NUMBER[^<]*)<\/td>/g;
  This one line contains all four pieces of data needed to complete the record, so if the page fails to parse, the first place that needs to look at is whether this still works.
  c) Do anymore format changing and then print it out in the same format as is accepted by the C++ code.
  3) If the page does not even pull down correctly anymore, time to goto the previous mentioned websites to determine if what happened. Possible problems are (a) Mapquest no longer supports that type of query or (b) Mapquest has changed the layout of the request or (c) Mapquest has gotten fancy and now requires more authenication or javascript or HTTPObjects.
  (a) This type of problem is a major problem because the demo system needs it so some redesigning of what the system is capable of is needed. Goodluck
  (b) This shouldn't be too difficult to determine. Just type garbage into the new HTML box such as the following link:
  http://www.mapquest.com/maps/map.adp?formtype=searchadv&searchtype=search&country=US&addtohistory=&cat=AAABBBCCC&address=DDDEEEFFF&city=GGGHHHIII&state=ZZ&zipcode=55555
  and try and determine what data goes where in the new query. If the requesting layout changed, then most likely the display layout changed so then this reduces to a (2) problem.
  (c) This is a major problem that either requires additional perl modules or major rewriting or perhaps even a language change. If this happens, perhaps there is a legecy Mapquest website that will stay on the web or there is another site that gives directions in a layout similar to Mapquest that can be used. I know of none but I also am not looking since Mapquest works wonderfully for my purposes.
  Hints:
  - Do not remove all html initially because
  a) It is quite difficult to remove all HTML using just a regular expression.
  b) If the HTML is removed, context of the data is lost. Sure you can look for "miles away" and the specific layout of a phone number, but you can do that while keeping all the HTML present.
  - It may help to split the document into lines by newlines or
  s.
  a) On the other hand, you may split the pieces of HTML you may be wanting to reg'ex find, so be careful.
  - Download as many HTMLs as you can to find a pattern. It is not fun to find a pattern and spend time coding it only to find out that it won't work for all of them.
  - Do realize that you may not be able to get all the entries. Some entries may not have "miles away" written by them, so they may not be easy to capture and not really worth the extra time.
  - Perl is a powerful reg'ex language and the current coder does not have tons of experience with perl. Therefore there are probably much easier\better ways of writing what is there. Don't follow the bad practices if there is a better way of writing what is already there.
  - Some perl expressions result in the rest of the lines becoming comments so sometimes things like #" or # '"])> must be written after a reg'ex so that emacs will behave and color things correctly.