sql查詢中如何去除某個字段重復(fù)的數(shù)據(jù)？可以優(yōu)化，我們最近在做的是用了3臺30G內(nèi)存，12核心的虛擬機。對于6億1千4百萬的數(shù)據(jù)（Mysql中是130G，parquet格式是30G），執(zhí)行多字段分組聚

sql查詢中如何去除某個字段重復(fù)的數(shù)據(jù)？

可以優(yōu)化，我們最近在做的是用了3臺30G內(nèi)存，12核心的虛擬機。對于6億1千4百萬的數(shù)據(jù)（Mysql中是130G，parquet格式是30G），執(zhí)行多字段分組聚合 某字段count(distinct)耗時在30秒左右，而對于普通查詢耗時在2秒內(nèi)。所以我覺得對于你的情況，優(yōu)化空間還很大

用SQL語句怎么過濾重復(fù)數(shù)據(jù)？

1、查找表中多余的重復(fù)記錄，重復(fù)記錄是根據(jù)單個字段（peopleId）來判斷復(fù)制代碼代碼如下:select * from peoplewhere peopleId in (select peopleId from people group by peopleId having count(peopleId) > 1)2、刪除表中多余的重復(fù)記錄，重復(fù)記錄是根據(jù)單個字段（peopleId）來判斷，只留有rowid最小的記錄復(fù)制代碼代碼如下:delete from people where peopleId in (select peopleId from people group by peopleId having count(peopleId) > 1)and rowid not in (select min(rowid) from people group by peopleId having count(peopleId)>1)