Importing Posts to WP using API

Below is the code I am using to pull data from an API to import it into my WordPress database. Getting the data from the API works fine and everything is gravy on api calls that return fewer results. However, with this particular function it is returning just over 9000 and breaking during the foreach loop that goes through each returned element; sometimes after 2500, sometimes after 5000. Also, as you can see I set the max execution time for PHP to 5 minutes but it stops just after a minute and doesn’t return an error.

This ones driving me crazy! Any help would be greatly appreciated.

Cheers


/**
 * Import vehicles
 */
function n4s_import_vehicles() {

	ini_set('max_execution_time', 300);

	//Initialize curl
	$ch = curl_init();

	curl_setopt($ch, CURLOPT_URL, 'http://api.curtmfg.com/v2/GetVehicle?year=&make=&model=&style=&dataType=JSON');
	curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	$ch_data = curl_exec($ch);
	curl_close($ch);

	if(!empty($ch_data)) {

		$i = 0;

		$json_data = json_decode($ch_data, true); //print_r(count($json_data)); exit();
		foreach($json_data as $data) {

			$post_title = $data['year'] . ' ' . $data['make'] . ' ' . $data['model'] . ' ' . $data['style'];

			$post = array(
				'post_type' => 'vehicle',
				'post_status' => 'publish',
				'post_title' => $post_title
			);

			$post_id = wp_insert_post($post);

			$vehicle_year = str_replace('.', '-', $data['year']);
			$vehicle_make = str_replace(' ', '-', strtolower($data['make']));

			if(!empty($post_id)) {

				// Update Year
				wp_set_object_terms($post_id, $vehicle_year, 'vehicle_year');

				// Update Make
				wp_set_object_terms($post_id, $vehicle_make, 'make');

				// Update Model
				update_post_meta($post_id, 'n4s-vehicle-model', $data['model']);

				// Update Style
				update_post_meta($post_id, 'n4s-vehicle-style', $data['style']);
			}

			$i++;

			if($i == 1000) {
				$wpdb->queries = array();
				$wp_actions = array();
				wp_cache_flush();
			}
		}
	}
}

Looking at the code I wonder if there is a way to save the data in chunks. From the looks of it at the very least 36,000 individual queries are going to be issued and that is a significant problem regardless of whether it works or not. I’m not familiar with WP but one would hope there is a way to update multiple rows/entities using a single query rather than issuing separate queries for every post. If not that is something your going to need to write because the way your doing it right now is going to be prone to failure not only short term but also long term as the data set scales.

Yeah, you’re definitely right. I guess I’ll head back to the WordPress codex and see if I missed something. If not, I’ll have to write some custom functions.

Thanks