How to fetch data from any website using esp8266
How to Fetch data from any website using ESP8266 | IoT Projects
Timecodes:
hello youtubers welcome to my next
tutorial on how to fetch data from any
website you want so this is the Sunday
morning seven Feb 2016 in India time is
10:30 5 a.m. and it's the winter morning
so let's see you what's the temperature
in my city so I am press the temperature
button so it is processing some data and
the temperature here is around 22 degree
Celsius blue it's not that much cold so
yeah so this is all we got to make today
we will learn how to fetch data from any
website so let's get started so
basically I have used this touchscreen
module so that it kept easy for you to
understand this project but you can use
anything you can use any LCD or
7-segment display to display the integer
numbers anything you want yes you
required is a Wi-Fi module okay so first
of all first of all let's declare the
less discuss the theory behind this or
how this process is working then we will
come to the code so basically let's take
example of live time in India
okay so life time India okay so I will
go to one of the website okay let us say
time and date calm okay so this is the
website and this is the current time of
India let's go this HTML page inspect
so basically when we request this link
or this URL it gives it in response this
much or this code in response we get
this code now from whole this or this
long code we want only the data of the
current temperature so we want something
something to request or something which
we request and in response we get only
the time only the required data so for
example your time is again inspect
where's the time time time time where is
the time I'm going to select this and
again inspect so this division okay so
here is the time now we want to target
this particular data so for that we
require an API now API stands for the
application program interface in detail
if you want to study about API you can
watch another videos but for those who
want overview of what an API does for
that I will explain API is basically
interface through by using API we can
target the required data from whole HTML
page and we can fetch or we can get the
data required data only from the whole
page available now the question is how
to make an API now API can be made
manually and for those who don't know
about coding like me can go to this
great website called things big calm
things be calm is great grid website for
starting into getting into the IOT
Internet of Things it's basically a
framework so it will automatically
provide us the required API any from any
website offer on any server so let's see
what's the procedure of making an API so
first of all you need to make an account
I already logged into my account so we
will go to apps apps and we will go to
the last block things HTTP or thing as
TTP simplify device communication with
web services and API is so here this
thing we will require so we will make a
new thing HTTP now the now we need to
provide some of the information that
from which website we want and for which
data from the whole page we want we need
to provide some basic data only and it
will automatically generate an API so
this one we will name that API suppose
five time India okay so now we will do
we have to provide the URL so we will
provide the URL of the website from
which we need to fetch the data so copy
and paste the URL so this is the sec
step authentication username password
leave it blank and we need to get SST
HTTP is all about getting and posting so
here we need to get the data so we'll
select the get and the version of HTTP
is 1.0 1.1 and the last and most
important front waters thing is parsed
train now what is that we need to
provide a tag or what you can say huh a
tag or an identity of the data which we
want to fetch so here is the time so we
will right click the time and copy XPath
XPath will copy the tag like something
or ID yes it will fetch the ID of that
data from that page so we will provide
this ID to pass string okay so this was
a pound of data we need to give and it
will automatically automatically
generate an API key this is an API key
particularly for this for fetching the
lifetime anyone having this key can face
that lifetime from that website our time
indeed.com
so we will save this thing as TDP let's
see the miracle of API so this is the
API and this is a get request we need to
simply post this link on to our browser
and let's see what's happened okay
so rather than the whole HTML page we
caught the required data that is the
time of lifetime in India that is 10
41:38 a.m. so this is the base of this
project making an API indent becomes a
very simple most simplest using this
thing's pick calm I'm sorry I am NOT
going into detail about API because the
tutorial will be
more longer so you can watch the API
what what we'll learn about API on
anywhere from YouTube or Wikipedia so I
explain the basic task of API to fetch
the link or to face the data from the
website okay so we have already
generated the API now let's see how to
use that API through esp8266 now let's
come back to the code okay so up till
here is the necessary declaration of
this TFT module which is necessary to be
declared before using this and this is
the host name and password
for this we need to provide for
connecting our esp8266 to a Wi-Fi
network so my host name is SMS and my
password is 8 times F then the functions
need to be performed
so I already perform the functions like
gold and temperature and I don't think
when I use I've also performed string
use yes
so this is basically the collection of
the functions now let's start or let's
start with a new function name string
time let's say we will make a function
of time and which will return a string
in response
so this or our declaration and setup I
have what you can say the baud rate is
set to 9000 sorry around 5 to double 0
and this is required to initialize that
watchman touch screen and the buttons
function this function will print the
buttons onto the screen for those who
have used this touchscreen module will
get easily understand so this all are
the buttons to be printed on the screen
so let's make a new button name a time
so I will just copy here we need to
provide the coordinates and the size of
the rectangle so I am simply copying the
size and changing the coordinates to 180
and this string or the data to time and
that this is yellow and I think the data
is sufficient ok let's upload and see up
with this a button should be appear
should appear on the screen naming name
time ok so ok ok ok I need to change the
coordinate here also so it will be 90
180 okay set cursor I need to change to
200 the y-axis is 200 okay so okay so
button name time appears now next after
buttons let us see what is there okay
after button a school goes into the loop
until here are the necessary coordinates
or 2 for the touch touch coordinates ok
so anywhere we whenever we touch the
screen it will go inside this function
and the coordinates will be saved in
integer X and integer Y so coordinate
and it will be show the coordinates
would be shown by the this speed or
sorry X&Y on our serial monitor we can
see the coordinates and according to the
coordinates we need to assign the
coordinates for particular function for
example for the reset button the
coordinates of X are 320 from 226 to 320
and for wife is from 82 to 129 so
likewise we need to assign the
coordinates so whenever we touch in a
row in this limit it will automatically
call a function reset same way we need
to make the coordinates for time I've
already assigned the coordinates for
time and we need to just copy and paste
another function because the pro the
what is at the command remains the same
only the name change this so we can just
simply copy and here it will black out
the rectangle above the buttons so it
will basically the clear what is printed
before and if deeper then it will show
time whenever it is touched I'm sorry
time and cursor zero comma zero wide
exercise three and fetching data and
call a function named time instead of
views but this is this much amount of
functions or this much command will be
executed whenever we touch in this
coordinates that is the time button so
now we need to just make a spring or
come I'm sorry we need to make a
function named time so we will make a
function here the spring time okay so
here also I will simply copy and paste
because the procedure is same changes
only in the API key only in the API
and the name so I will copy up till here
copy and paste it in this function okay
so I will discuss the instructions one
by one okay okay so before before that
we need to see what happens when we
press the reset button of each of which
commands are executed when you press
this reset button so I have made a
function Carol to reset I will discuss
that okay said he said so it will first
of all reset this Wi-Fi module with
80-plus rst and will disconnect any
disconnect the Wi-Fi through which is
all which the Wi-Fi module is already
connected so it will disconnect Wi-Fi is
connected and it will select the mode 3
that is both access point and station
mode and it will search for the Wi-Fi
network and will connect with the
network whose host name and password you
have provided so let's see what happens
when you press the reset button so what
he said ok so it shows resetting ok mode
with is equal to 3 and searching for
Wi-Fi network it will show connected
when it connects through the network so
stream is provided mmm come on come on
ok so this all our functions when
performed when the reset button is
pressed so after connecting connected
after the device get connected to the
Wi-Fi now the now we need to start from
here ok so will after connection
we will stop after connecting to the
network we will establish a TCP
connection so for fetching the data we
need to first of all establish a
connection through a command it T plus C
IP start so we need to send a T plus C
IP star is equal to the type of
connection it is TCP connection the host
server name and the port number port
number is here of course ad as it is
through HTTP ok okay so we can provide a
host sir
were named now the host server name is
basically the name of that host who is
aap isolated here it is API dot thanks
fake calm so I have made the string ATC
F star is equal to inverted comma TC be
inverted comma close comma inverted
comma if they are good things become
inverted form ax comma 18 and slash
lesson for enter
I am sending this string through send
data function of society data function
requires requires the string the timeout
period is 3000 that is 3 seconds and 0
and 1 if it is 1 it will show the
response on to a serial monitor that is
for debugging so I return 0 year so
after sorry as soon as we send this
command it will automatically ask us to
say ask us to provide the number of
characters we need to send which is 80
plus CI percent is equal to 90 so here
we are sending a 90 characters after
this this command so we need to provide
number of characters here and this will
be remained constant because in all the
different or different different API is
from things be calm only the API key
will be changed and the rest of all
things will remain same so the character
also will remain same as the key is
constant of 16 characters I think okay
so so this also remains constant the
change is only in this key now keep
generated is whichever key is undeterred
if you had things to do okay so we need
to just copy the key copy and paste the
key your control V okay so this is a now
we need to change the string to the IME
time okay so it will send this string
first of all the get API API key and the
host server name that is AP I thought
things picked odd form okay let's see
I'm just commenting this steps to filter
the data because before that you must
but must be aware that
which datum we are getting and why we
need to filter so we'll just make it one
so it shows the response in the season
monitor now I think control you upload
ok need to change here T IME time and
upload ok so let's see what data we are
getting and why we need to filter that
data ok so as soon as I press the time
but only source to fetching data ok ok
so here is the receive data are now
receive data that is IP D comma 11 comma
11 18 13 am closed ok out of that we
will require only this much amount of
data so we need to filter this data and
rest of the data are not at all useful
to us we need to filter it so that's why
I returned the code for filtering the
data and this is also very much very
much easy let's let's see the logic
behind this filtering so every time
whenever it gives response it will show
it will use plus IP D comma the number
of characters in response and then
semicolon and the data will start so we
required the data after the semicolon
and the data before this closed that is
the C character so I have made a
algorithm which will filter from a
filter the data starting from vertical
character and ending at particular
character so here our data is starting
from a semicolon and ending before
ending at C of C for closed so character
before require data starts and collected
after end require data ends ok so every
time is not the case that the data is in
between semicolon and the closed
function it may be between any of the
characters so you just need to change
the character before the starting of the
data and after the end our
after end of the theta so just provide
the characters and the data will be
filtered now let's see if this works or
not let's upload this okay so let's see
when I press the time button
it must works hope it works
okay so required data
we got the required data that is along
25 0 9 a.m. the delay is because of the
my screen recorder pressed so that is
the delay so any of the required data or
the required task is completed we face
the data from the website called time
and date calm and displayed on to our
touchscreen module or you can use the
data anywhere you want you can make even
a Wi-Fi of Wi-Fi clock which has the
data every 10 minutes or every second
every 5 second or 10 second and you want
so this is all about precision you can
fetch any data from any website like the
stock market data the Live Gold rate
which I have done like temperature
lifetime like YouTube views anything you
want let's make an API and making API
becomes much much much simpler using
things big calm so explore that website
the great website for IOT related things
and stay tuned for next such amazing
tutorials until then goodbye thanks for
watching
