• Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • About
  • Life
  • Tech
  • Travel
  • Work
  • Questions
  • Contact

Welcome

.

How to handle huge json over network call through pagination in java?

April 10, 2020 by

Questions › How to handle huge json over network call through pagination in java?
0
Vote Up
Vote Down
Garmaine asked 3 years ago

I have this existing API running in loop which creates a connection and fetches huge json(1 GB) in each iteration from a service through restTemplate as follow:

ResponseEntity<String> response = restTemplate.exchange(uri.get().toString(), HttpMethod.POST,entity,
                    String.class);

The response is then converted to a complex java object through GSON. The problem with above approach is, rest template converts the inputstream to String through StringBuffer which ends up creating lots of char[] eventually running out of memory (OOM) when the loop is iterated for too long which usually is the case. In place of RestEntity, I even used HttpClient, it too does the same (expand char array).

To solve OOM issue, I refactored the API to stream the data to file. Creating a temp file in each iteration and converting to objects as follow:

File targetFile = new File("somepath\\response.tmp");
FileUtils.copyInputStreamToFile(response.getEntity().getContent(), targetFile);
List<Object> objects = gson.fromJson(reader, new TypeToken<List<Object>>(){}.getType());

is this the way to go or is there any effective approach to solve such problems? Maybe pooling connections instead of creating new in each iteration (will this be considerable change?)

Also, on analysing the API on 4GB xmx, 2GB xms heap, jvisual shows the below:

enter image description here

As it can be seen, the running thread has been allocated huge bytes.

Heap size during the API runtime:

enter image description here

Are you looking for the answer?
Original Question and Possible Answers can be found on `http://stackoverflow.com`

Question Tags: java, json, out-of-memory, stream, stringbuffer

Please login or Register to submit your answer




Primary Sidebar

Tags

Advancements best Business strategies commercial convenience economic Finances Cognitive decline Financial growth firm Future Hidden Gems Home hydration Impact Innovations lighting line of work Mental health Must-See New York City office patronage Productivity profession Profitability tips Profit optimization pursuit recreation Revenue enhancement romance sippy cups social station Technological breakthroughs technology toddlers trading transaction Treasures Uncover undertaking Well-being Wonders Work Young onset dementia

Newsletter

Complete the form below, and we'll send you all the latest news.

Footer

Footer Funnies

Who knew that reading the footer could be such a hilarious adventure? As we navigate websites, books, and documents, we often stumble upon the unassuming space at the bottom, only to discover a treasure trove of amusement. In this side-splitting compilation, we present 100 jokes that celebrate the unsung hero of content – the footer. Get ready to chuckle, giggle, and maybe even snort as we dive into the world of footnotes, disclaimers, and hidden comedic gems. Brace yourself for a wild ride through the footer!

Recent

  • Unveiling the Enigma: Almost-Magical Lamp Lights Highway Turns
  • The Impact of Young Onset Dementia on Employment and Finances: Optimizing Post-Diagnostic Approaches
  • 11 Wonders of 2023 Technological Breakthrough – Unveiling the Future
  • Work from Home and Stay Mentally Sane – Achieve Productivity and Well-being
  • Hidden Gems of New York City – Uncover the Must-See Treasures!

Search

Tags

Advancements best Business strategies commercial convenience economic Finances Cognitive decline Financial growth firm Future Hidden Gems Home hydration Impact Innovations lighting line of work Mental health Must-See New York City office patronage Productivity profession Profitability tips Profit optimization pursuit recreation Revenue enhancement romance sippy cups social station Technological breakthroughs technology toddlers trading transaction Treasures Uncover undertaking Well-being Wonders Work Young onset dementia

Copyright © 2023