Dataset

다양한 종류의 데이터셋에 대한 소개와 이를 수집하는 방식을 알아봅니다!

Ⅰ. Dataset

하나의 완전한 데이터셋을 소개합니다.

• AwesomeKorean Data

  • https://github.com/songys/AwesomeKorean_Data

• KorQuad : 한국어 Machine Reading Comprehension 데이터셋

  • https://github.com/korquad/korquad.github.io

• CounselGPT : 한국어 심리 상담 데이터셋

  • https://github.com/MrBananaHuman/CounselGPT

• Welfare-QA : 한국어 복지제도 관련 QA 데이터셋

  • https://github.com/ash-hun/Welfare-QA

• KLUE

  • https://github.com/KLUE-benchmark/KLUE

• Korpora

  • https://github.com/ko-nlp/Korpora

• 국립국어원 사전

  • https://github.com/spellcheck-ko/korean-dict-nikl

• 한국어 챗봇데이터

  • https://github.com/songys/Chatbot_data

• Korean HateSpeech Data

  • https://github.com/kocohub/korean-hate-speech

• Ko-Wiki-Data

  • https://huggingface.co/datasets/maywell/ko_wikidata_QA

• Kor-Wiki-Hop

  • https://huggingface.co/datasets/KETI-AIR/kor_wiki_hop

• Korean Human Judgement

  • https://huggingface.co/datasets/HAERAE-HUB/Korean-Human-Judgements

• Ko-Calibration

  • https://huggingface.co/datasets/maywell/ko-calibration

Ⅱ. Data Collecting

주어진 문서에서 내용을 추출하거나 외부에서 데이터를 수집/크롤링하는 레퍼런스를 소개합니다.

• MegaParse

  • https://github.com/QuivrHQ/MegaParse
• FireCrawl

  • https://github.com/mendableai/firecrawl
• Playwright

  • https://github.com/microsoft/playwright
• MinerU

  • https://github.com/opendatalab/MinerU
• PDFMathTranslate

  • https://github.com/Byaidu/PDFMathTranslate
• OpenParse : Easy PDF Chunking open-source library

  • https://github.com/Filimoa/open-parse

Ⅲ. Data Engineering

• Airflow

  • https://airflow.apache.org/
• Prefect

  • https://www.prefect.io/