Parsing a twitter feed

Get Started
1

I need some things clarified about how to parse a twitter feed:

am writing a little twitter search app now… here are my steps:

  1. do search:
$tweetSearch = $TwitterConn->get('search/tweets.json?q='.$searchForThis.'&count=3');  
	//	(var $searchForThis comes from the request in pretty conventional fashion..)

get result as JSON:

	$TwitterConn->decode_json = false;

now this “JSON” is not really a JSON – it looks like a JSON, but technically it’s a string, yes?
(meaning: I cannot grab it with jQuery getJSON() method, right? for that it has to technically reside in a .json file? not sure about this…)

so this is exactly the feed I get from the twitter get() method I mentioned above…
(man, this looks so messy, can’t even tell where one record ends and the next one begins (this one is three records, or entries…) but it validates as JSON…:wink:

  1. so in Javascript to “convert” this string to an object (so I can loop through it) I do:
 	var searchResults = <?php echo $tweetSearch?>;

does this make sense?

(I don’t know how to create objects and loop through them in PHP…:wink: that would have to be something like a Hash Map in Java, yes?)

  1. then, to loop through it, I use the standard jQuery method to loop thru objects:
$.each(<?php echo $searchResults?>, function(propName,propValue) {....}	

but this line throws an [FONT=Courier New]'undefined'[/FONT]....  on that embedded PHP variable...

so what am I doing wrong? am I on the right track in general? I find this Twitter API to be quite challenging actually… the Sitepoint article fretburner posted on a previous thread about this they even talk about the “convoluted” Twitter instructions for their API…:wink:

thank you…

2

Hey,

There’s no problem using $.getJSON with a PHP script that returns JSON as its output. Here’s an example script, which builds upon the stuff we talked about in your previous threads, that you can adapt for your own purposes:

index.html


<!DOCTYPE html>
<html>
  <head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8">
  <title></title>
  <script src="http://ajax.aspnetcdn.com/ajax/jquery/jquery-1.10.1.min.js"></script>
</head>
  <body>
    <form method="post" action=''>
      <input type='text' name='terms'>
      <input type='submit' value='Search'>
    </form>
    <script>
      $('form').submit(function(e){
        e.preventDefault();
        var terms = $("input[name='terms']").val();
        $.getJSON('search.php', {terms: terms}, function(response){
          console.log(response.statuses);
        });
      });
    </script>
  </body>
</html>

search.php


<?php
require_once('twitteroauth/twitteroauth.php');

$terms = filter_input(INPUT_GET, 'terms', FILTER_SANITIZE_ENCODED);

if ($terms)
{
    $twitter = new TwitterOAuth($consumerkey, $consumersecret, $accesstoken, $accesstokensecret);
    $twitter->decode_json = false;

    $tweets = $twitter->get("search/tweets.json?q=$terms&result_type=recent&count=10&include_entities=true");
}
else
{
    $tweets = '{}';
}

header('Content-type: application/json');
echo $tweets;
3

thank you fretburner…

for some reason I can’t get anything to print on the page…

 $('form').submit(function(e){
        e.preventDefault();
        var terms = $("input[name='terms']").val();
        $.getJSON('search.php', {terms: terms}, function(response){
          console.log(response.statuses);    // prints
         $('div#console1').append(response.statuses);   //  doesn't print...
         $('div#console1').append(response);   //  doesn't print...
        });
      });

(console.log() stmt prints fine… but the append() methods are being ignored…)

none of this, which I put inside getJSON() method, prints either:

  $.each(response.statuses, function(propName,propValue) {	
		  		
			if (propName === 'text') {
				$('div#console2').append(propName + ' : ' + propValue + '<br>');
			}
				
			if (propName === 'created_at') {
				$('div#console2').append(propName + ' : ' + propValue + '<br>');
			}
			
			if (propName === 'lang') {
				$('div#console2').append(propName + ' : ' + propValue + '<br>');
			}
		  });

and I don’t get any JS errors…

thank you…

4

if I do the submission the “old-fashioned” way, i.e., w/o AJAX, then the php just prints this:

{ }

like no data at all is being returned…

5

The reason this isn’t working is that response.statuses is an array of objects, so when you’re passing it to $.each(), propName is the array index and propValue is the tweet object.

Try this:


<form method="post" action=''>
    <input type='text' name='terms'>
    <input type='submit' value='Search'>
</form>
<ul id="results"></div>
<script>
    (function(){
        var terms = $("input[name='terms']").val(),
            results = $("#results");
        $('form').submit(function(e){
            e.preventDefault();
            $.getJSON('search.php', {terms: terms}, function(response){
                $.each(response.statuses, function(index, tweet) {	
                    results.append('<li>' + tweet.text + '</li>');
                });
            });
        });
    })();
</script>
6

thank you fretburner, but am I not looping through these objects here?

 $.each(response.statuses, function(propName,propValue) { ...... }

isn’t this the same as your

$.each(response.statuses, function(index, tweet) { .....}

with diff var names?
(jQuery knows whether it’s an obj or an array, and assigns that first var accordingly, right?? (an index if it’s an array, a string if it’s an object… or not?? :wink:

either way it still doesn’t work…

this is my entire code now…


[FONT=Courier New]
<!DOCTYPE html>
<html>
<head>
<script src="http://ajax.aspnetcdn.com/ajax/jquery/jquery-1.10.1.min.js"></script>
</head>
<body>

	<form method="post" action="">
		<input type="text" name="terms">
		<input type="submit" value="search">
	</form>
	
	<div id="console1"></div>

	<script>		
		(function(){
			var terms = $("input[name='terms']").val();
			$('form').submit(function(e) {
				e.preventDefault();
				$.getJSON('search.php', {terms: terms}, function(response) {
					$.each(response.statuses, function(index, tweet) {	
		          		     console.log(response.statuses); 
					     $('#console1').append(tweet.text + '<br>');
					});
				});
			});
		})();
	</script>

</body>
</html>[/FONT]

nothing prints on the page… (console.log() stmt prints fine…)

???

thank you…

7

I changed the variable names to make it clearer what I was doing within the loop. Your original code was written to iterate over the key/value pairs of a single object, but as I explained response.statuses is an array of objects, so when you tried to do this:

if (propName === 'text') {
    $('div#console2').append(propName + ' : ' + propValue + '<br>');
}

you wouldn’t get any output, because propName would never contain any of the object keys… it would contain array keys (i.e. 0, 1, 2 etc.)

As for why your current code doesn’t work, I pasted it into my test file and I was getting an error in the console saying that e was undefined. I changed the version of jQuery to 1.10.2 and that seems to have solved the problem, and now the code outputs the statuses to the page.

8

thank you fretburner…

oh brother… well, mine doesn’t work with either jQ 1.8 or 1.10… and I get no errors on e object…
but I see what you mean… I get confused sometimes, when looping thru an array of objects…:wink:

also, I’m curious as to why you wrapped the function in “()” and all that…:wink: (that’s like plugin code…:wink:

also: weird… am getting this error in Chrome only… (1.8 & 1.10…)

GET http://localhost/tw/jquery.min.map 404 (Not Found)

WTF???

thank you…

9

so this would be the correct way to loop?

$.getJSON('sp_process.php', {terms: terms}, function(response) {
	$.each(response.statuses, function(index, tweet) {
	
		$.each(tweet,function(propName,propValue) {
			
			if (propName === 'text') {
				console.log(propName + ' : ' + propValue);
				$('#console1').append(propName + ' : ' + propValue + '<br>');
			}
				
			if (propName === 'created_at') {
				console.log(propName + ' : ' + propValue);
				$('#console1').append(propName + ' : ' + propValue + '<br>');
			}

		});
	});
});

still nothing prints… and I get that weird error in Chrome… :frowning:

10

The idea of wrapping all the code in a function like that is that then everything inside the function executes in a separate scope and variables (in this case, just terms) don’t pollute the global scope. It’s called an IIFE (immediately-invoked function expression). It’s not really needed in this example, but it’s a good habit to get into.

As for the errors, I’m not sure what might be causing them - certainly your latest code example should work OK. I’ll have look into it.

11

Have a look at this page: http://stackoverflow.com/questions/18365315/jquerys-jquery-1-10-2-min-map-is-triggering-a-404-not-found - it has instructions about how to get rid of the error message.

Could you put your code online somewhere I could test it? When I reproduce your code on my machine, it works fine.

12

so now I’m getting that ‘e’ error, even though am using jQ 1.10…

oh brother… this is frustrating…

ok, fretburner… here’s all my code…
http://www.mayacove.com/dev/tw/tw_api.zip

it just contains four files:
– search.html
– search_results.php
– and the two includes…

very much appreciate your help.,

thank you…

13

Right, I think I’ve finally gotten to the bottom of the problem… you need to make the following changes:

Edit all of the PHP files and remove any empty lines and space before the opening <?php tag - it should be the very first thing at the top of the file, otherwise the script will throw a ‘headers already sent’ error.

Edit your HTML file and move the terms variable declaration:

(function(){
    // Move from here..
    $('form').submit(function(e) {
        e.preventDefault();
        var terms = $("input[name='terms']").val();  // ..to here
        // rest of code not shown

this one is my fault… although I’m surprised it didn’t cause me problems when I was testing on my laptop :confused:

14

oh man… finally!!! finally the feed is printing on my page!!
thank you fretburner…:slight_smile:

only thing is, the key-value pairs don’t print in the same order as I place them in my code… my code:


$.each(tweet,function(propName,propValue) {							
	
	if (propName === 'text') {
		$('#console1').append(propName + ' : ' + propValue + '<br>');
	}
	
	
	if (propName === 'user') {
		
		$.each(propValue,function(userPropName,userPropValue) {
			
			if (userPropName === 'screen_name') {
				$('#console1').append('user : ' + userPropValue + '<br>');
			}
		});
	}
	
	
	
	if (propName === 'created_at') {
		$('#console1').append(propName + ' : ' + propValue + '<br>');
		$('#console1').append('<br>');
	}

});

what I get:

created_at : Wed Aug 28 23:35:21 +0000 2013

text : ‘In My Place’ ~ Coldplay 
user : TheNewSeanKeany
created_at : Wed Aug 28 23:35:19 +0000 2013

text : ‘Coldplay- the scientist’ hermosa canción!
user : CaamilaAlegre
created_at : Wed Aug 28 23:35:12 +0000 2013

text : Yellow by Coldplay has some of the prettiest gutiar ever. It’s so beautiful, and I just want to sit outside while its raining and play it.
user : twirlyenough

PS: there are no urls for individual tweets? I couldn’t find tweet urls in this JSON feed…

thanks again, FB…:~)

15

The order of object properties in JS is not fixed and varies across browsers. If you want to output properties in a certain order, you can change your code (see my example below).

You can create the URL for an individual tweet using the returned data:

var tweetUrl = 'http://twitter.com/' + tweet.user.screen_name + '/status/' + tweet.id_str;

Here’s the modified JS that outputs the properties in the order you want, and with a link to the original tweet:

(function(){
    $('form').submit(function(e) {
        e.preventDefault();
        var terms = $("input[name='terms']").val();
        $.getJSON('search_results.php', {terms: terms}, function(response) {
            $.each(response.statuses, function(index, tweet) {
                $('#console1').append('text : ' + tweet.text + '<br>' 
                                    + 'user : ' + tweet.user.screen_name + '<br>' 
                                    + 'created_at : ' + tweet.created_at + '<br>'
                                    + '<a href="http://twitter.com/' + tweet.user.screen_name
                                    + '/status/' + tweet.id_str + '">Original tweet</a><br><br>');
            });
        });
    });
})();
16

awesome… thank you very very much for all your help and your patience, fretburner… I really appreciate it…

this thread has gotten very long… if I have another question I guess I’ll start another thread…

have a great day… thanks again…

17

Have to ask got a copy of the full code that’s working I wanted to try grab all tweets with in a set time frame and import them to a database but could only get one result every time due to the nutty array

18

[FONT=Verdana]the “nutty” array… lol… that “array” you can get as JSON by using this line…

 $twitter->decode_json = false;

right after this line:

 $twitter = new TwitterOAuth($consumerkey, $consumersecret, $accesstoken, $accesstokensecret);

this is what I use to get the tweets:

 $tweets = $twitter->get('statuses/user_timeline.json?screen_name=NewYorker&count=1');

(you don’t need an absolute url here, even if you’re on a localhost, the twitter API ads the rest…)

so this returns a JSON with one record, you copy that entire thing and put it in JSON lint and it will format it for you…(JSON Lint is a very useful JSON-validating tool, and it has the added bonus that it formats the JSON neatly, so it won’t look like a “nutty array” anymore…:wink:
(what I do is I copy this newly-formatted json and save it as a .json so I have it handy when I’m parsing it…)

to grab tweets within a set frame I suppose you can use the “created_at” property in that “nutty array”…;~)
(i.e., the JSON…)

[/FONT]

19

[FONT=Verdana]
PS: I meant “within a set **time frame…”

[/FONT]