101. Talking to the internet

This post is about swapping information with the internet, including how to post web forms (which is the really new information in this post).

Downloading from the internet

I covered the downloading of images in post #62, here. I apologise for covering some of the same material again, but if you’re like me, that may help make it clearer for you.

You can download a web page, or image, to Codea using http.request. The big catch is that the internet may take a while to respond, so Codea doesn’t wait for an answer. It carries on with the next line of code.

There are two obvious questions.

1. what happens when the web page (or image) does download?

2. what if I want Codea to wait for the download to complete?

.

Codea allows you to decide what to do when the page downloads, because the http.request command needs the name of a “callback” function that will be run when the download completes. If you’re not sure what callbacks are, then I’ve confused you further – so perhaps a simple example will make things clear.

Downloading an image

Suppose we want to download an image, and put it in a variable called img. When you tell Codea to download it, using http.request, Codea will do that, and will then move on to the next line of your code, without waiting for the download to complete. It will keep drawing the screen 60 times a second, too. So when the download does complete, how are you going to know, and how are you going to tell Codea what to do with it?

Codea has a solution. It basically says “give me the name of a function when you request a download, and I’ll run it when the download is complete, and I’ll give you the downloaded information as the first parameter. You can use that function to do what you want with the download.”

So we write a function

--the data parameter on the next line 
--contains the image data
function ImageReceived(data)
    img=data
end

and then we ask Codea to do the download (obviously you’d use a real URL)

    http.request(url,ImageReceived)

So the sequence is

1. you request the download

2. Codea starts downloading, but keeps running your code

3. when it completes, ImageReceived runs

.
Pausing Codea while the download finishes

Suppose you need the image to download before doing any drawing. Maybe it’s the background image, for example. So perhaps you want to put a “Please Wait” message on the screen while it’s downloading, and when it’s done, you’ll start drawing normally.

The easiest way to do this is with a “state” variable, and do our drawing depending on its value. So we could start off with (say) state=”downloading” in the setup function. In the ImageReceived function, we then set state =”ready”. Then, in the draw function, we draw the wait message if state=”downloading”, and do normal drawing if state=”ready”, using an if test.

This pausing process is explained in some detail in my post 62 referenced above, so I won’t go through all that again.

Finally, what happens if the download fails? Well, Codea lets you specify an extra function name to run if that happens, so they thought of everything. Here’s an example

    http.request(url,ImageReceived,ImageFailed)

Scraping the internet

You can use Codea to pull useful information out of web pages. For example, you could extract weather information from a weather service. If you use http.request to retrieve the weather forecast, you will get the HTML text of the web page. It may be so complex you can’t make sense of it, but maybe, you can find what you need and figure out a way to search for it that will always work (eg it is between the text xxxxx and yyyyy).

Sending information to the internet

URL parameters
There are two usual ways of sending personalised information to a web page. One is to include it as part of the URL. If you’ve ever seen a URL with lots of stuff on the end like “language=en&style=brief&x=3”, these are all personalised instructions for the web page (the separator is an ampersand &). They work best for short text, obviously.

If you need to do this, then you have to figure out which items to add to the url and what values to give them. Note they can be in any order, though.

Web forms
Codea can also post web forms for you. Any time you’ve had to fill in an online form, chances are that Codea could submit it for you, because forms are turned into text strings before being sent to websites, so it’s possible to “fake” – ie automate – web form results without actually filling a form in online.

I’ll give you an example first, then explain it.

    headers={}
    headers["Accept"]="text/plain"
    headers["Content-Type"]="application/x-www-form-urlencoded"
    headers["Accept-Charset"]="utf-8"
    data="field1="..value1.."&field2="..value2.."&field3="..value3"  
    http.request("URL",SubmitSuccess,SubmitFailure,
                {method="POST",headers=headers,data=data})

Don’t worry about the first few header lines. They will rarely change. The data line stores all the form field information, as explained below. Finally, the request is sent, with functions to run if it is successful or a failure, and an extra little table containing the form information. So if it completes in this case, SubmitSuccess will be run, and you’ll be able to play with whatever data you have received in return.

The form fields are listed in any order in the form name=value, separated by & (ampersand). Don’t put quotes around text strings. So this kind of data string is fine

    data="name=Ben&Age=24&Address1=6 Palmer St&Address2=Perth"

How do you know what the form fields are? What I do is open up the web page containing the form, on my PC (or mac) and use View Source Code to open a new window with the raw HTML code (what this option is called and where it is in the menus will depend on your browser, but it isn’t in mobile safari, so don’t try this on the iPad).

Then I search for the text string <form , which is the start of the form. This tag usually contains the URL that will be called when the form is submitted, and you will need that. Then you have to find all the input textboxes, checkboxes, lists or whatever are in the form, until you hit which tells you the form is finished. For each input item, you need two things. It should have a name, and a value, and you have to string them together as data. There are often input items which are marked as hidden (ie invisible normally) and used to pass extra information to the website. Don’t forget to include these.

If a website is feeling unfriendly, it may include a “session variable”, a random number or string that is different each time the web page loads. This will make it impossible to submit a form without actually loading the webform page, something we don’t want to do.

Codea’s internet capability

So Codea has (in my view) provided a very neat and concise way of talking to the internet. It’s not simple – except perhaps for images – but that’s not Codea’s fault. It’s difficult in most programming languages, because the internet is complex.

I don’t want to go much further, because I would bore most of you, but if you are keen, I will share more of what I know. Just message me as ignatz on the Codea forum.

Advertisement