Remove duplicate lines from a file in Scala
How to remove duplicate lines from csv or txt file?
duplicatefilejavascalaDevelopment tips
How to remove duplicate lines from csv or txt file?
The answer is quite straightforward: you basically need BufferedReader and BufferedWriter, and this also works for large files quite well.
def removeDuplicatesFromFile(fileName : String) { val reader = new BufferedReader(new FileReader(fileName)) val lines = new mutable.HashSetString var line: String = null while ({line = reader.readLine; line != null}) { lines.add(line) } reader.close
val writer = new BufferedWriter(new FileWriter(fileName)) for (unique <- lines) { writer.write(unique) writer.newLine() } writer.close }def removeDuplicatesFromFile(fileName : String) { val reader = new BufferedReader(new FileReader(fileName)) val lines = new mutable.HashSetString var line: String = null while ({line = reader.readLine; line != null}) { lines.add(line) } reader.close
val writer = new BufferedWriter(new FileWriter(fileName)) for (unique <- lines) { writer.write(unique) writer.newLine() } writer.close }
By adminduplicatefilejavascala
Recent Posts
-
Top 5 Business Success Metrics You Should Be MonitoringSeptember 11, 2025
-
5 Tips To Build a Data-driven CompanySeptember 9, 2025
-
Looking for a technical co-founder? You won’t succeed without reading this.September 5, 2025
Archive
Tags
#CloudTechnologies #Code #CyberWhale #GenerativeAI Analytics bi Big Data Business Business growth business intelligence Cambrian Explosion Cloud data Data Capture Data Management Data Science Data Stack Data Synchronization Data Systems Data Warehouse DBLog deeplearning4j development dev tools DoorDash ELT ETL Hacks java jobs libs machine learning parsing Production python request Retail SaaS scala Strategy tip tutorial Uber word2vec xml
AI Strategy & Consulting
Build a future-ready AI strategy with trusted guidance